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A  Contemporary  Example  on 

the  Need  for  Better  VV&A 
Approaches 


The  JSF  Program  Office  initiated  a 
roadmap  for  the  verification, 
validation,  and  accreditation  (VV&A)  of 
the  labs  and  models  intended  to 
become  test  venues,  per  the  mid¬ 
course  risk  reduction  strategy  of  2007. 

The  roadmap  serves  as  a  gauge  to 
measure  the  contractor’s  progress  in 
completing  the  accreditation  support 
packages  needed  before  success 
criteria  can  be  resolved  using  the 
models. 

The  current  roadmap  indicates  that  50 
percent  of  models  will  be  accredited 
during  the  final  year  of  flight  testing, 
an  approach  with  substantial  risk. 
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Improved  Decision  Support  Quality 
through  a  Balanced  Approach 
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Acquisition  Community-led 
VV&A  High  Level  Task  (HLT)  Summaries 


V-AQ-2:  “Risk  Based  Methodology  for  Verification,  Validation  and 
Accreditation  (VV&A)”  The  degree  of  VV&A  required  is  explicitly  tied  to 
both  M&S  use  and  the  user  risk  incurred  if  the  M&S  does  not  provide 
accurate  results.  A  methodology  that  tailors  VV&A  planning  and 
implementation  based  on  known  risk  factors  will  provide  a  framework  in 
which  VV&A  implementation  trade-offs  can  be  made,  information/fidelity 
requirements  can  be  assessed,  and  a  VV&A  cost  model  can  be 
developed. 

V-C-2:  “Improving  VV&A  Implementation”  Increase  VV&A 
implementation  and  enhance  M&S  credibility  by  transforming  VV&A 
practices  from  current  subjective  methods  into  objective  examples  or  use 
cases.  Explore  emerging  technologies,  standards,  and  applicable 
methods  that  could  be  applied  to  reduce  costs,  schedule,  and  improve 
reuse. 


SERC  Annual  Review 
Nov  2010  Page-4 


Verification,  Validation,  &  Accreditation 


(M&S  PE  Funded  V-C-2  &  V-AQ-2  ) 


FY:  09-10  FY:  10-11 
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Risk-Based  VV&A 

(M&S  PE  Funded  V-AQ-2) 


The  purpose  of  this  effort  is  to: 


Balancing  the  cost  of 
knowing  against  the  risk  of 
assuming. 


>  Improve  Department  of 
Defense  (DoD)  Modeling  and 
Simulation  (M&S)  Verification, 
Validation,  and  Accreditation 
(VV&A)  by  establishing  an 
effective  risk-based 
methodology  for  VV&A. 

Since  the  cost  of  verifying  and 
validating  simulations  is  often 
high,  V&V  investments  should 
be  weighed  against  the  risk  of 
making  a  bad  decision  because 
of  unreliable  M&S  results. 
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JHU/APL  VV&A  Implementation: 
Ihi?J|  Defining  &  Addressing  Gaps 

(M&S  PE  Funded  V-C-2) 


Improving  W&A  Implementation: 
Defining  and  Addressing  W&A  Gaps  IPR 


The  purpose  of  the  effort  is  to: 

>  Support  the  identification  of  VV&A 
gap  areas  that  inhibit  effective 
implementation  of  VV&A  processes; 

>  Develop  supplemental  capability 
required  to  implement  the  Risk-based 
Accreditation  (RBA)  methodology; 
and 

>  Evolve  technology,  in  the  form  of 
automated  tools  and  metadata 
specifications,  that  increase  VV&A 
implementation  efficiencies. 
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Systems  Engineering  Research  Center  (SERC) 

VV&A  Topic 

(M&S  PE  Funded  V-C-2) 


Proposed  SERC  research  topic: 
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Explore  &  promote  a  deeper 
understanding  of  emerging  &  current 
approaches,  technologies,  or  practices 
underway  within  the  academic 
community  &  gain  a  broader  perspective 
on  their  potential  applicability  to  the  M&S 
community  V&V  needs 

The  19  SERC  university  partners 
generated  six  responses,  with  selection 
of  two  proposals  for  implementation 
lasting  6  to  12  months 


Two  performers: 

Georgia  Institute  of  Technology 
University  of  Alabama  Huntsville 
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Georgia  Institute  of  Technology 

VV&A  Study 

(M&S  PE  Funded  V-C-2) 


Representing  System  Models 

With  SysML  Unified,  Connected.  Consistent,  Explicit 


operational  concert 

!■> 


system  moae 


spreadsheet 


The  purpose  of  this  SERC  effort : 

>  Demonstrate  how  to  address 
many  of  the  gaps  identified  in  this 
VV&A  Research  Topic  by  applying 
SysML  and  MBE/MBSE  technology 


Answers  to  Questions  *  II 

Who  cares? 

—  All  M&S  and  VV&A  stakeholders  (given  benefits  below) 

If  you're  successful,  what  difference  will  it  make? 

—  Our  approach  provides  the  Enabling  Capabilities  seen  in  the  table  rows 
below,  which  produce  the  main  Primary  Impacts  in  the  columns 

—  Ex.  Related  earlier  studies 
achieved  75%  reduction 
in  M&S  time  and  enabled 
increased  analysis  intensity 

—  We  will  endeavor  to  demo 
and  quantify  similar  benefits 
in  this  SERC  effort 


Primary  impacts 
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>  Engage  practitioners  in  the  DoD 
ecosystem  to  prioritize  needs  and 
to  test  resulting  concepts 

>  Apply  concepts  in  one  or  more 
key  DoD  test  beds 
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University  of  Alabama  at  Huntsville 

VV&A  Study 

(M&S  PE  Funded  V-C-2) 


White  Paper 

Verification,  Validation  and  Accreditation  Shortfalls  for 
Modeling  and  Simulation 

Introduction 

This  white  paper  is  a  proposal  to  research  and  explore  the  use  of  Architecture  Analysis 
and  Design  Language  AADL  and  tools  designed  with  AADL  for  the  verification, 
validation  and  accreditation  of  complex  systems.  This  research  work  will  be  done  in 
partnership  with  the  System  Architecture  Vutual  Integration  (SAM)  consortium.  The 
SAM  is  a  five  year  endeavor  and  is  presently  in  its  second  year  of  research  in  W&A  for 
complex  systems. 

Background 

The  Rotorcraft  Systems  Engineering  and  Simulation  Center  (RSESC)  at  the  University  of 
Alabama  in  Huntsville  has  been  working  with  the  Army's  Aviation  Engineering 
Directorate  (AED),  Software  Engineering  Directorate  (S ED),  Redstone  Test  Center 
(RTC),PEO- Aviation,  the  Project  Management  Offices  under  PEO- Aviation  and  several 
other  Army  PEOs  for  many  years  assisting  to  solve  m  the  areas  of  systems 
engineering  and  system  design  problems.  Over  the  last  six  months.  RSESC  has  been 
working  with  AED  and  SED  on  the  on-going  issue  of  how  to  verify  and  validate  complex 
systems.  One  of  die  issues  is  moving  from  document- centric  requirements  to  a 
mathematical  model  to  verify  and  validate  the  requirements  on  complex  systems. 
Presendy  there  is  not  a  way  to  predict  the  performance  of  complex  systems  all  the  way 
through  to  integration.  Software  and  system  design  languages  are  loosely  defined  and 
therefore  do  not  provide  the  precise  definition  needed  for  high  fidelity  simulation  and 
quantitative  modeling  and  formal  methods.  When  considering  analysis  tools  there  is  a 
limitation  in  their  capability  to  work  together;  therefore  problems  are  typically  found 
after  the  systems  are  built.  It  is  believed  that  an  architectural  context  is  needed  to  resolve 
this  issue.  By  having  a  high-level  specification  tool  or  architecture  to  perform  a  virtual 
simulation  or  complete  system  analysis  ensure  system 

success  before  system  integration  and  test. 

Presendy  if  semantics  are  incorrecdy  applied,  developers  no  longer  have  a  common 
understanding  of  the  language  for  integration  of  architectural  models,  and  must  be 
experts  in  all  domains  where  the  model  is  expected  to  be  used.  Another  option  that  is 
used  is  utilizing  a  custom  language  which  leads  to  a  need  to  define  it  well  and  document 
it  well.  Then  upkeep  and  revalidation  of  assumptions  becomes  an  issue  each  time  models 
are  integrated  that  have  their  own  semantics  or  the  entire  model  must  be  upgraded. 


Rotorcrqfi  Systems  Engineering  and  Simulation  Center 


The  purpose  of  this  SERC  effort : 

>  Focus  on  verification  &  validation 
(V&V)  of  complex  systems 

>  Identification  of  existing  useful  tools 


Leverages  experience  working  the 
government  customers  in  the  Army  Aviation 
Directorate,  Software  Engineering 
Directorate,  Redstone  Test  Center,  and  PEO- 
Aviation. 

Will  benefit  from  insights  resulting  from 
Systems  Architecture  Virtual  Integration 
(SAVI),  an  international  industry  consortium 
involving  Boeing,  Airbus,  Lockheed  Martin, 
BAE,  FAA,  SEI,  etc 
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SIMAF  VV&A  of  a  Live,  Virtual,  Constructive 
(LVC)  Environment  Use  Case 

(M&S  PE  Funded  V-C-2) 


Net-Enabled  Weapons 


Fix  and  Track  Taraet 


iracK  larg 

te  sensors) 


Network 

Architecture 


C2  Node  Assesses  Situation  & 
Determines/Direct  Engagement  Actions 


Develop  a  Use  Case  for  VV&A  of  a  LVC  Distributed 
Simulation  Environment  (DSE): 

>  Document  lessons  learned  from  prior  VV&A 
efforts  related  to  LVC  DSEs. 

>  Document  applications  of  IEEE  std 
1516.4TM-2007;  MIL-STD-3022  &  Risk  Based 
Approach  for  VV&A 

>  Consolidate  documented  best  practices 
methodology  for  establishing  acceptability 
criteria  for  DSE. 


Joint  Command  and  Control  Joint  Control  For  Net-Enabled  Weapons 
from  7  March  2007  Col  Richard  W.  Leibach,  USAF  Director  JT&E 
presentation 


Linked  live -virtual 
constructive 


- -  Sensor 


Scalable,  multi  - 
resolution  models 


r-iecj  Strike  Platform 


_  O 

'--Target** 


Persistent,  reusable,  interoperable 
composabte  ,  Integrated 


figixm  1.  HoortKorts&an  of  data  flow  in  a  «rija'  orvuonmert. 

Tim  Menke  &  W.  Walter  March,  ITEA  Journal  2009;  30:  469-472 


>  Using  the  methodology;  plan,  conduct,  & 
document  a  “use  case”  for  VV&A  of  a  LVC 
DSE. 

>  Insert  another  accredited  simulation  into 
the  LVC  DSE  to  re-verify,  re-validate,  &  re¬ 
accredit  the  DSE. 

>  Recommend  best  practices  for  VV&A  of  a 
LVC  DSE  in  a  final  report. 
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Domain-relevant  “Best  of  Breed”  M&S 
Tool  Suite  VV&A  NAVAIR  Use  Case 

(M&S  PE  Funded  V-C-2) 
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Accreditation  (VV&A)  Process 
Developed  By  The 
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Battlespace  Modeling  V&V  Branch  (navair  5  4.2.3) 


James  N.  Elele,  PhD 
Branch  Head 
NAVAIR  5.4.23 


Battlespace  Modeling 
Verification  &  validation  Branch 


EMAIL:  james.elele@navy.mil 
Phone:301-342-4154 


* 


Apprwea  nr  punc  re*ea&e;  asruutan 
tsirtmuar 
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Multiple  communities  within  the 
Department  of  Defense  are  interested  in 
accelerating  schedules,  improving 
credibility,  and  reducing  costs  of  technical 
execution  of  M&S  support  to  decision¬ 
makers  through  tailored  accreditation 
support 

The  scope  of  this  effort  was  to: 

>  Bring  knowledge  of  existing  VV&A 
“best  practices”  to  funded 
implementation  of  credibility 
assessments  of  high-priority  DoD 
M&S  applications; 

>  Examine  the  practicality  of 
leveraging  Service-specific  VV&A  for 
domain-relevant  Department-wide  tool 
suite  applications;  and 

>  Identify  Service-specific  needs  & 
estimate  both  cost-avoidance  & 
incremental  costs  for  plan  to  address. 
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VV&A  Program  Management  & 

Integration 

(M&S  PE  Funded  V-C-2) 


AGOUISH  ION  MODELING  A  SIMULATION  WORKING  GROUP 
VERIFICATION,  VALIDATION,  AND  ACCREDITATION  SUBCOMMITTEE 
TFRMS  OF  REFERENCE 
May  14.  2010 

Purpose 

Thnw!  tsrmc  C4  n?f®re^cc  deserts  ihopuipaiue  and  stineUr*  of  th<  V/cnrictf  to  • 
Va-daWcr,  a  no  Acc^ecMation  <W&A1  Subcommittee  T>«  subcommittee  •*  subordiuaiu 
1o  the  Aocjtrtiban  MAS  Won t»>q  Croup  (AMSWC'  T he  -^rms  e*  reference  document  the 
subcommittee'*  acoar-  stakeholders,  ia*»3  lesponaHjlllLea,  and  ye'teaa  admvwl/abve 
function*  and  ynli  r*o«;»o  *aM  uni  i  rancj-iled  by  the  AMSWG 

Scope 

Contnouhofw  horn  Hie  W&A  Subcommittee  »v»»  wupooT  Acted  4-3 1*  tha  Acquedlinri 
MAS  Master  ^too  (AMSM^I  anc  Goal  3  of  the  M&o  Gommttees  Strateg  c 

Vbiuii  for  DoD  M&S 

•  AMSAIP  Action  4-6 

o  Foy®'  coet  cnect-ve  W&A 

•  Strategic  Vision  ler  DoC  M&S  rGoal  3 1 

o  Provide  management  oroceeQos  foe  mooes,  ynuialions,  end  associated 
U**tu  that  ncluoe  pracucai  guideline*  for  h  e  *  W&A 

The  VVftA  Subcommittee  will 

•  Coordinate  with  I *v?  AM5WG  Chai»mari  on  proposed  fespoi^ee  !o  Conpregsonai 
maur.ds  "cgarding  W&A  of  roooals  eimulatk>««  and  assorted  data  used  in 
the  acquiEft'on  proofs 

•  Be  tho  Acquisition  Coimumly  conduit  for  sharing  W&A  lAfAfbMbon  by 

pi  ousting  a  virtual  VV&A  hrs-n  Irusl  for  the  Acquisition  Ctnim.W.y 
r^sgonoff^j  to  request?  for  W&A  frfnrniaticin  fmm  Ar.i;u«il*jo  Community 
members 

u  eerebliamiM;  web  accea*  to  pertinent  W&A  information  ro*ovanl  to  irwi 

Acquisition,  Commusty 

•  Conduct  penodK  technical  interchange  mooting*  to: 

-  promos  the  importance  of  W&A 
make  W&A  i*  formation  ava  M6e 

r>-  the  ‘go  to*  place  vxoere  W&A  tifornation  *  shared  arvl  evcnonoM 
Kjerrtt y  Acquisition  Gommuniy  members  W&A  needy  ana  issues 

•  Help  evo‘v«?  !he  practice  of  W&A  by 

Revu?*»ng  end  cornmnnlvg  nn  W&A  “iu'JIbh 

Prnird^  input  tn  it*.  rnvww  vni  upduu  uf  W&A  pc^cy  and  standards 

Membership 

Participation  is  votirtsry  Mernoorship  will  corrp^n  g^vyrn'wem  oganzahons  engaged 
m  W(VA  atfPMta*  ('mrinm«nh;t-on  policy,  guidane#  tttMtaaMv  etc  f 


The  VV&A  Subcommittee  is  a  coalition  of 
the  willing. 

Help  evolve  the  practice  of  VV&A  by: 

>  Reviewing  and  commenting  on 
VV&A  studies 

>  Providing  input  to  the  review  and 
update  of  VV&A  policy  and 
standards 

Supported  by: 

>  ITEA  LVC  Sub-committee 

>  “Gray  Hair”  Support  to  Key 
Activities 
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Key  Acquisition  M&S 
Contact  Information 


Acquisition  Community  Steering  Committee  Representative 

>  Kristen  Baldwin,  DDR&E/SE/SA 

(703)  681-7000,  Kristen. Baldwin@osd. mil 

Acquisition  Modeling  &  Simulation  Working  Group  Chair 

>  Philomena  Zimmerman,  DDR&E/SE/SA 

(703)  681-6544;  Philomena.Zimmerman@osd.mil 

Acquisition  M&S  Team 

>  Crash  Konwin,  BAH  Support  to  DDR&E/SE/SA 
(703)  681-6576;  Kenneth. Konwin. ctr(g>osd. mil 

>  Marcy  Stutzman,  NGC  Support  to  DDR&E/SE/SA 
(301)  317-9698;  Marcy.Stutzman@ngc.com 
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Systems  Engineering: 
Critical  to  Program  Success 


Innovation,  Speed,  and  Agility 

http://www.  acq.  osd.  mil/se 
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